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The Education Endowment Foundation (EEF) is an independent grant-making charity dedicated to 
breaking the link between family income and educational achievement, ensuring that children from all 
backgrounds can fulfil their potential and make the most of their talents. 


The EEF aims to raise the attainment of children facing disadvantage by: 


e Identifying promising educational innovations that address the needs of disadvantaged 
children in primary and secondary schools in England; 


e Evaluating these innovations to extend and secure the evidence on what works and can be 
made to work at scale; 


e Encouraging schools, government, charities, and others to apply evidence and adopt 
innovations found to be effective. 


The EEF was established in 2011 by the Sutton Trust, as lead charity in partnership with Impetus 
Trust (now part of Impetus-The Private Equity Foundation) and received a founding £125m grant from 
the Department for Education. 


Together, the EEF and Sutton Trust are the government-designated What Works Centre for improving 
education outcomes for school-aged children. 


This project was co-funded by the EEF and Nominet Trust as part of a funding round focused on 
digital technology. 
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Executive summary 


The project 


The Visible Classroom project aimed to use lesson transcripts to promote effective teaching practice 
and improve the attainment of pupils in primary school. The approach used ‘real-time captioning’ 
technology to generate a live transcript of teachers’ speech in lessons and was developed by the 
University of Melbourne and technology provider Ai-Media UK. 


Lesson transcripts were made available to teachers and used as the basis of a personalised 
‘dashboard’ which provided information about key features of lessons, such as the balance of teacher 
and pupil talk, the amount of ‘thinking time’ given to pupils and the types of questioning used. 


In addition, live transcripts could be projected directly onto a whiteboard or tablets. Pupils could use 
the transcripts to review learning instructions and goals, and used the devices to provide feedback on 
their learning at the end of each lesson. 


This pilot evaluation involved ten primary schools in London and the West Midlands. Participating 
schools received training and followed the approach over two terms in the academic year 2013-14. 


The evaluation had three aims. First, to assess the feasibility of the technology and overall approach. 
Second, to provide recommendations that could be used to improve the approach in the future. Third, 
to assess the promise of the approach, and its components, to inform any future trial. 


This project was co-funded by the EEF and Nominet Trust as part of a funding round focused on 
digital technology. 


Key conclusions 


Overall, teachers were positive about the Visible Classroom approach, and believed that it 
had the potential to benefit both themselves and their pupils. 


Most teachers were adept at using the technology in the classroom, even if they had not 
done so before this trial. There were some technical problems related to hardware, software, 
and internet connections, but after an initial bedding-in period most were overcome. 


Though few teachers spent time reviewing the verbatim transcripts, the online dashboard 
and more detailed feedback reports based on the transcripts were seen as valuable tools to 
support teacher development. To maximise the impact of the feedback, teachers would 
benefit from being given greater opportunity to review and discuss their practice with peers 
and managers. 


Pupils did not seem to use live transcripts of teacher dialogue regularly, consistently, or in a 
way that would suggest an obvious benefit in learning. Teachers had mixed views on 
whether the live transcripts might have additional benefit for disadvantaged pupils or their 
peers. 


Further research would be required to assess the level of impact the approach has on 
academic attainment. Prior to considering a full trial it would be valuable to undertake some 
additional development work to refine the approach. 
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What did the pilot find? 


The pilot study found that the Visible Classroom approach is feasible and has the potential to make 
an impact on teacher practice that may lead to benefits in pupils’ learning. 


Most schools that participated in the project were enthusiastic about the approach. Teachers 
appreciated having a record of their teaching performance and believed that they benefited from the 
analysis dashboard and the more detailed feedback reports. Though very few teachers spent time 
reviewing the entire transcripts of their lessons, all participating teachers used the dashboard 
regularly, and felt that this was the aspect of the approach most likely to change their behaviour and 
lead to increased pupil attainment. To further improve the impact of this component, it would be 
valuable to ensure that teachers are provided with allocated time to reflect on feedback, and are able 
to discuss the feedback with peers or managers. Most teachers were adept at using the technology, 
even if they had not done so before the project. 


There was less evidence that live transcripts of lessons directly benefited pupils by increasing their 
engagement or enabling them to review previous parts of the lesson. Though some teachers thought 
that the transcripts were of benefit to children, many classes did not follow the live transcripts and 
some teachers actively discouraged pupils from following the live transcript during whole-class 
teaching. 


Teachers also had mixed views about which pupils were likely to have most benefited from the live 
transcripts. Some felt that transcripts offered reinforcement and information checking for poorer 
performing pupils whilst others felt the pupils had to be fairly skilled to find the information they 
needed and so only the higher performing children were able to do this. 


How was the pilot conducted? 


The pilot was a qualitative evaluation of the Visible Classroom approach based on 27 semi-structured 
interviews with staff in participating schools, and two classroom observation visits. During 
observations the evaluation team assessed how the approach was used, levels of pupil engagement, 
and the use of tablets. 


How much does it cost? 

The approach is estimated to cost approximately £367 per pupil, based on five hours captioning per 

week over one school year and class sizes of 25 pupils. This estimate includes analysis costs (£158 

per pupil), equipment (£12 per pupil), training (£125 per teacher), and live captioning (£192 per pupil). 
Question aaiare | tare} Comment 


All schools completed the project. There were 
no major technological problems. 


Was the approach feasible? Yes 


Most teachers were very positive about the 
approach, and believed it had the potential to 
change their practice. However, pupils did not 
seem to use the live transcripts in a way that 
would suggest an obvious learning benefit. 


Is there evidence of 
promise? 


Prior to considering a full trial it would be 
Is the appreaen feady 10ra No valuable to undertake further development work 
full trial? ; 

to refine the delivery of the approach. 
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Introduction 


Intervention 


The University of Melbourne and technology provider Ai-Media UK developed the concept of using 
captioning with pupils in Years 3-6 within the classroom setting to support pupil attainment and 
develop teachers’ practice. Hattie’s (2009) Visible Learning work provided a major impetus for giving 
teachers feedback about their teaching practice. He proposed that when teaching and learning are 
‘visible’-—that is, when it is clear what teachers are teaching and what students are learning—student 
achievement increases. When teachers become more aware of their practice and gather structured 
feedback on their performance, they can make appropriate changes to better support pupils’ learning. 
This project aimed to make teaching and learning ‘visible’ by using real-time captioning. They called 
this approach ‘The Visible Classroom’. The University of Melbourne and Ai-Media UK worked with 
Nesta as their UK-based delivery partner for the EEF-funded development stage of this project. 


While speaking in class, teachers wore a microphone which relayed their speech back to a transcriber 
or ‘captioner’. This person (based in the UK) typed up what the teacher said, producing a live 
transcript, similar to subtitles. This written text was transmitted back to the class so pupils were 
provided with a real-time display of the teacher’s dialogue during lessons (‘captioning’). This could be 
displayed (using specific software) on tablets shared amongst the class. 


Figure 1: The live transcription process 


i 
captioner Q 


Text to user 
within 7 seconds 


Each transcript was read by a transcript coder, part of the programme team. These coders measured 
aspects of the teachers’ practice against a 16-point rubric developed by the programme team. This 
data was fed into the teacher’s online dashboard. 


After the lesson, teachers could review their teaching using the full transcripts and the online 
dashboard. This summarised the key features of their lessons, such as balance of teacher and 
student talk, ‘thinking time’ given to pupils, the types of questioning used, and speed of speech, along 
with the pupil feedback. Pupils were also required to complete an online survey about their 
experience of each lesson (see Appendix 3). This was completed jointly by the group of pupils sharing 
each tablet. 


In order to further develop teacher involvement, an online forum was set up for participants and 
regular newsletters were circulated to those taking part. An individualised feedback report (see 
Appendix 4) was sent to each teacher twice during the trial (after six weeks for teachers in Phases 1 
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and 2 and after two weeks for teachers in Phase 3), which gave more detailed insight into their 
practice over the preceding period. 


More information about the approach is provided at: http://www.nesta.org.uk/project/digital- 
education/visible-classroom#sthash.75MlceDw.dpuf 


Background 


According to the intervention team, the premise behind the intervention is that ‘teachers who make a 
difference to student attainment understand their level of influence and have the ability and resources 
to collect and review evidence to support this impact’ (Hattie, 2009; Clinton, 2015). 


The approach had previously been used in a trial involving pupils with low academic achievement, 
pupils affected with learning difficulties, and pupils with who were deaf or hard of hearing (Melbourne 
Graduate School of Education, 2014). This evaluation instead focussed on implementing the 
approach for whole classes, assessing how well the technology could fit into mainstream education in 
the UK, and whether it had the potential to help raise the attainment of disadvantaged pupils. 


Objectives 


The aim of this developmental pilot was to test the feasibility and acceptability of the delivery model 
for schools in England. The evaluation had three aims. First, to assess the feasibility of the technology 
and overall approach. Second, to provide recommendations that could be used to improve the 
approach in the future. Third, to assess the promise of the approach and its components, to inform 
any future trial. More specifically, it sought to answer the following questions: 


e Do any technological barriers exist and, if so, how can they be overcome? 

e What more generally are necessary conditions for success? 

e How well can the approach be embedded within the school setting and day-to-day teaching? 

e What are teacher attitudes to the approach? 

e What is core to the model and what is optional to the model? For example, could schools just 
use the transcripts, not the live captioning feed? 

e Other questions, such as: are there any specific aspects that may differentially impact on 
disadvantaged children? 

e How often should lessons should be captioned (both in terms of what is realistically 
deliverable and what is attractive to schools)? 


Project team 


NatCen Social Research led this evaluation, directed by Amy Skipp. Alexandra Fry, Nilufer Rahim, 
and Jonathan Paylor carried out interviews and observations of participants. Schools were recruited, 
trained, and managed by Nesta and the University of Melbourne team. 


Ethical review 


The project evaluation, including all materials used with teachers and data collection, was agreed by 
the internal ethical review board of NatCen. 
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Methodology 


Evaluation design 


A process evaluation was carried out spanning the full duration of the trial (from March to July 2014). 
Qualitative data from in-depth interviews with key stakeholders and observations of classroom 
practice were collected throughout each of the phases. 


NatCen was commissioned by EEF to evaluate the project during the development stage, feeding 
back findings to the intervention team on a regular basis so that they could make changes to the 
project and take into account participants’ experiences. 


The evaluators worked collaboratively with the intervention team in order to: 


e Collect process evaluation data to inform the design and technology used for any possible 
future trial. 

e Define a general theory of change behind the intervention. 

e Identify the hypothesised causal pathways that may lead to outcomes and the best way to set 
up and evaluate the intervention. 


Participants 


In total, ten primary schools were recruited to take part in the pilot study. The study was split into 
three overlapping phases with staggered start times in order for the approach to be adapted and 
developed: 


e Phase 1 involved five schools. Teachers from these schools participated in the project for up 
to four months during the Spring and Summer terms 2014. 

e Phase 2 involved three schools. Teachers from these schools participated for up to two and a 
half months during the Summer term 2014. 


In both Phases 1 and 2, during weeks 1-4 teachers were asked to use the approach in at least two 
literacy lessons a week. After this they were invited to use the approach however much they wanted, 
in whatever lessons they chose. 


e Phase 3 involved two schools and tested a more intensive version of this approach, in which 
they used the approach for at least one lesson a day (resulting in 5-10 hours per week) but 
over a period of just one month. Teachers from these schools participated for up to one 
month at the end of the Summer term 2014. 


Levels of use were captured by the intervention team from numbers of sessions booked and 
transcripts produced. 


The ten state schools that participated in the project were from two regions in England—the West 
Midlands and Greater London. A total of 36 teachers participated in the project. 
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Table 1: Characteristics of participating schools 


rsTed sTofo) | Phase Local authority Number of Pupils Number of 
area pupils receiving participating 
free school teachers 
meals at 
Yoq Vole) 7) 


1 
2 
3 
4 
5 
6 
7 
8 
9 


Hackney 
Hounslow 
England national average 


—_— 
oO 


The intervention team was responsible for recruitment. Schools within chosen local authorities were 
invited to take part in the pilot study and evaluation. Phase 1 schools were recruited at an event in 
Dudley. Phase 2 schools were largely recruited through Nesta contacts, with one school recruited at 
the initial recruitment event. Phase 3 schools were recruited through Nesta contacts. After 
recruitment, each school nominated teachers to participate. In most cases these were all of the 
teachers for a whole year group (such as all Year 5 teachers). Schools selected were intended to be 
representative of a range of potential users of the technology so that findings would be as 
generalisable as possible. This included schools in urban and rural areas, schools with higher and 
lower levels of technology experience, and those with more newly qualified teachers and those with 
more experienced teachers who may be less familiar with the use of technology. 


All schools were sent information packs and consent forms to be signed by all teachers involved in 
the project as well as information sheets for parents. Pupils’ speech was not captioned or recorded in 
the transcripts, so parental consent was not sought. 


The intervention 
This pilot study of the Visible Classroom approach ran during the academic year 2013-14. 
The intervention team: 


e provided schools with between six and eight tablets per class (unless the school already 
owned devices). 

e supplied the appropriate equipment and software to allow real-time captioning from 
participants’ classrooms. This included microphones and mobile telephone connection. 

e facilitated analysis of each transcribed lesson via an online dashboard (see Appendix 2). 

e produced two feedback reports per participant, based on analysis of several of their lessons. 

e set up an online forum for participants and circulated weekly e-newsletters containing project 
updates and examples of practice. 


The school participants: 


e used captioning for lessons as instructed with pupils in Years 3-6 
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e booked their transcription sessions with captioners in advance and uploaded any preparatory 
materials (to highlight particular vocabulary and topics and tasks, for example) 

e asked pupils to complete a brief survey of their experience of each lesson 

e received full transcripts of their captioned lessons 

e accessed an instant ‘dashboard’ of analysis of their lesson delivery 

e accessed an online discussion forum and were sent weekly newsletters about the trial. 


Theory of change 


The intervention team developed this model to reflect their approach prior to the pilot study. The 
evaluation was based around this, looking for evidence that the approach was leading to the outputs 
specified or that the outcomes detailed were being realised and by which mechanisms. The 
evaluation findings therefore could be used to refine this model or suggest where further evidence is 
required. 
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Figure 2: The theory of change 


‘Visible Classroom® Theory of Change 
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Methods 


Qualitative data from in-depth interviews with key stakeholders and observations of classroom 
practice was collected throughout each of the phases. 


At least one teacher was interviewed from every school. Teachers in each school in Phase 1 and 2 
were interviewed at least two (and up to four) times throughout the pilot study. Teachers in each 
school in Phase 3 were interviewed once, at the end of their phase. 


Table 2: Interviews carried out in each project phase 


Number of schools Number of interviews 


Two observation visits were carried out (one in a Phase 1 school, and one in a Phase 2 school) which 
involved watching how the intervention was running, interviewing staff, and holding small informal 
group discussions with pupils. More observations were not conducted due to timing and availability 
issues within schools and the evaluation team. 


All other interviews were by telephone and involved teachers updating us on progress, issues 
associated with implementation, and programme perceptions. 


The interviews were semi-structured, based around a topic guide (see Appendix 5) to ensure 
systematic coverage of key issues, but also flexible and interactive, allowing issues of relevance for 
individual respondents to be covered through detailed follow up questioning. 


When signing up to participate in the pilot study teachers were given details of what the evaluation 
would involve (see Appendix 6). They opted in to taking part and providing their contact details. 
Verbal consent was sought from interviewees at the start of the interview for their anonymised 
feedback to be reported on and shared with the intervention team. 


The interviews were digitally recorded and subsequently analysed using Framework, a systematic 
approach to qualitative data management developed by NatCen Social Research and now widely 
used in social policy research. All participants were told that everything discussed in the interview 
would remain confidential and would be treated in accordance with the Data Protection Act. In 
addition, it was made clear, both on recruitment materials and during the interview, that their views or 
opinions would not be discussed outside of the research team, including sharing individual feedback 
with the intervention team. 
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Process evaluation 


In this section, we examine each element of the Visible Classroom project in turn. Although each 
element is seen by the intervention team as an integral part of the approach, it was felt that it was 
important to investigate each separately. 


We then report on the overall intervention in terms of perceived impacts, future use and sustainability, 
and recommendations for potential future evaluation. 


Training 


This subsection covers the engagement of participating teachers and feedback on the training content 
and format. 


When the intervention team recruited schools to take part in the trial, it was often the headteacher 
who agreed for the school to participate in the project, although it was the teaching staff who would 
actually be involved in the day-to-day working of the pilot study. Therefore, several participants knew 
very little about the project and what participation would involve before the training event. They had in 
effect been told they were taking part and that they should attend the training. This may have had an 
effect on how ‘on board’ they were with the approach and their engagement and fidelity to the 
approach. 


A one-day training event was held for participants in each intake before the start of Phase 1 and 
Phase 2. For Phase 3, a two-hour training event was held with participants in their own school. 


Teachers found the training event very informative and were positive about the interactive approach. 
They had the opportunity to observe a demonstration of captioning and to access the system and see 
the feedback dashboard. They felt there was ample opportunity to ask the training staff questions. 
They appreciated meeting with staff from other schools to discuss the project and gather ideas of how 
to implement it. 


The training events were modified slightly after the first event (based on participant feedback) in order 
to provide sufficient theoretical background to the project and the concept of captioning, and to allow 
all participants sufficient opportunity to try out the technology. They also requested more explanation 
of the rationale behind which aspects of their practice were analysed and the evidence behind this. 


Teachers at the Phase 1 training seemed unaware of why specific aspects of practice were 
highlighted, or that they were going to receive more in-depth feedback throughout the trial. Once the 
pilot study was underway they were looking at the dashboard to see how many words a minute they 
were using but they did not know why this was important for their teaching practice or have any idea 
of benchmarking their practice against good practice. In order to self-evaluate they needed to have 
these principles firmly embedded in their minds so they knew what they were working towards. The 
evaluation team notified the programme team of this issue, which led to them changing the training 
content for Phases 2 and 3. 


Teachers requested that more choice of training dates be available to them. This was to ensure all of 
the staff involved had a chance to attend but that schools weren't left short-staffed by too many 
teachers having to be out on the same day. 


The format and content of the training day by Phase 3 were generally considered appropriate by 
attendees. 
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Technology 


This subsection covers: 


participants’ experience of technology use 

use of technical support 

provision of equipment 

reliability 

concerns over appropriate use of tablets by pupils 
accessibility 

suitable numbers of tablets per pupils. 


Participating teachers had a wide range of experience with using technology (including owning iPads 
or tablets, using laptops in class, and experience with online booking and support tools). Those few 
who were not proficient users of such systems struggled more with the concept of the project as well 
as integrating it into their everyday teaching and development practice. 


What also seemed to make a difference was that some schools had technical support available to 
them. Where this was the case, support was available for setting up the tablets ready for lessons, 
closing down and deleting apps after lessons, and dealing with any access problems and other 
technical issues throughout the project. For schools where such a person did not exist, teachers 
found that these tasks took quite some time, and that this time took them away from other tasks. 
Teachers who struggled with the technology also turned to other staff who had more personal 
experience for help, or to Ai-Media UK. 


The support from the intervention team was considered good, although response to issues did 
sometimes take some time. In a few cases this meant that schools were without one, two, or 
sometimes three out of eight tablets at a time. With too many tablets out of use, some teachers 
decided to suspend their involvement from the trial until the problems were resolved. 


Unfortunately some technological difficulties required tablets being replaced or being sent away for 
repair. Other issues appear to have been caused by incorrect use or lack of knowledge of the 
software and so should have been possible to remedy quite quickly, with sufficient and accessible 
support. 


In terms of equipment, early on it was suggested that teachers needed microphone clips and phone 
holders (so they did not have to hold a microphone or mobile throughout lessons) and these were 
provided. Similarly, checks were needed to ensure that any schools using different tablets (such as 
iPads they already had) were able to run all of the systems as they should. This caused some delay 
for some schools. Additionally, stands were ordered for the tablets to sit on as otherwise they were 
being laid flat on the table (where no one could see the screen content) or pupils were having to pile 
books up behind in order to stand them up. 


Many of the schools raised issues of reliability. It is not possible to know if this was due to problems 
with their use or problems with their equipment. When these issues occurred teachers reported that 
they generally ‘just put the tablets aside and went back to normal to prevent too much distraction 
from the lesson. This is a key consideration for any future trial as it could reduce the number of 
lessons that could be captioned. 


Use of the tablets and familiarisation with the software, dashboard, and making online bookings 
definitely bedded in over time. Teachers suggested having a two-week preparation phase before the 
trial begins to accommodate this familiarisation and to deal with any teething issues. 


Teachers reported initial concerns about pupils downloading apps and using the tablets in a way they 
should not. However, this resolved itself after a short time, as pupils lost interest in the novelty of the 


Education Endowment Foundation 14 


The Visible Classroom 


tablets. Some schools had felt the need to discuss appropriate use of tablets with participating pupils, 
but this did not appear to be any sort of problem throughout the trial. Many schools already used such 
technology in class and so giving tablets to children of this age was not considered inappropriate. 


Concerns were raised about the number of tablets available to pupils. In most classes, five or six 
pupils shared one 7” screen tablet. In many classroom settings, this meant that it was not possible for 
all of the pupils to see the transcript on screen. Many classes for pupils this age arrange seating in 
blocks and teachers said they felt it would work best if pupils could have one tablet between two. This 
would not only allow them to all be able to see the screen better but also increase the opportunities 
each pupil could take control of the tablet. In our classroom observations we saw that some children 
not only could not see a tablet screen from where they were sitting but also that many did not get a 
chance to hold the tablet, look through the transcript, or complete the lesson survey. This was clearly 
causing difficulties in some classes; for example, one interviewee scored the lesson poorly as he had 
not had a chance to touch the tablet screen in the lesson. 


Some children were given their own tablet because of special educational needs and/or behaviour 
issues. We saw an example of a magnified screen for a child with visual impairment and a few 
children who had their own tablets either because they sat separately in class or with a teaching 
assistant or because their behaviour meant they were unable to share. It is interesting to consider 
whether use of the tablets is meant to improve teamwork and sharing and therefore whether guidance 
should be provided on this. 


Likewise, some schools where several teachers were taking part in the trial were not provided with 
enough tablets for more than one class to use them at a time. This meant that teachers had to 
negotiate between them when each class could use the tablets. For schools where the whole year 
group does similar lessons at the same time, this led to some planning issues. Teachers suggested 
the optimum number of tablets in a class to be one between two. 


Some teachers were concerned about the high level of daily use of the tablets and how quickly they 
were looking damaged. There were slight concerns about how quickly the tablets would look tatty or 
be damaged and the question of whether protective cases could be provided to reduce this. 


In general, the technological issues were resolved and there were far fewer examples of issues after 
Phase 1. 


Captioning in class 


This subsection covers the use of live captions by pupils in class, referring back to transcripts, 
benefits of captioning for groups of pupils, the use of captioning in different subjects, and the extent to 
which the tablets caused distractions that may have detracted from learning. 


By captioning teacher’s speech in real-time and presenting it in written format almost instantaneously 
to pupils, it was intended that pupils would have several routes to learning, and thus have a direct 
positive impact on learning. The captioner aims to enable pupils to: 


e connect oral and written expressions of language 

e catch up on areas of the lesson they might have missed 

e review teacher instructions and explanations during lessons by looking back through the 
transcripts on their tablets. 


All teachers told us how important they considered it for pupils to be looking at the teacher when they 
are speaking to the whole class. This was a skill that pupils generally had to learn and it helped the 
teachers to control behaviour and engagement in learning. The idea of the pupils then having the 
option of watching the screen in front of them or interacting with the tablet while the teacher spoke 
was of some concern. All of the teachers spoke of needing each child’s attention and that the tablets 
were a potential distraction from this. 
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Initially pupils found the captioning novel. They were interested in the idea of someone sitting listening 
in to their class and then typing what was said. Some pupils noticed the delay between the teacher 
speaking and the words appearing on their screen and occasionally words being misheard or mis- 
typed. However, there were no reports of these being common issues. Some teachers instructed 
children to not look at the screens while the teacher was speaking. 


This novelty did not seem to last much past the first couple of weeks of the trial though. In our 
observation sessions (in week 3 of Phase 1 and week 7 in Phase 2) we only saw a few pupils even 
glance at the captions on screen during the lessons for either whole class or group/individual work. 


If following the live captioning is expected of the children, and the intervention team think that access 
to this written record is likely to have a beneficial impact on pupils, then the fact that there is very little 
use of it is a concern. The theory of change states that outcomes for pupils are achieved by: 


e Increased exposure to language 

e Increased access to the learning process, both in real-time via captioning, and potentially via 
the transcript available for review after the lesson 

e Ability to see and hear their learning 

e Asecond chance to engage with instruction. 


All of these points rely on the pupils using the captioning, and although it was not made explicit what 
extent of use is required, there was no evidence, from interviews or observations, that a high 
proportion of pupils were using the live captioning. It is possible that more instruction for the pupils on 
how to make the best use of captions would have been beneficial. 


In terms of their use, it was reported that some children scrolled back over what the teacher had said. 
In some cases teachers suggested this as a strategy if pupils missed something that was discussed, 
needed instructions repeating, or the teacher was explaining something to a small group. If the 
teacher felt they had said something to a small group which may benefit the whole class then the 
pupils would all be encouraged to look back through the record of what the teacher had said. 


In any future trial, the intervention team should make sure that the transcript is not used to the 
detriment of teachers’ practice. For example, if pupils are regularly not able to follow instructions there 
may be a risk that teachers keep telling them to refer to the transcript, rather than questioning if their 
initial explanations were sufficient. 


Teachers disagreed as to which students were more able to review and benefit from the transcripts. 
Some teachers thought that the ability to review what had been said would benefit lower ability pupils 
the most. They thought these children would be able to get the extra clarification they needed; for 
example, on what tasks they should be doing or how they should be doing them, and so would be 
more on-task and less likely to ask for help in class. However, other teachers thought that the ability 
to scroll back was dependent on the child having a certain level of ability in the first place. Teachers 
felt that finding the pertinent information or understanding it from the written word when they had not 
understood it from what was spoken required a high level of literacy ability. They therefore thought 
that the captioning was of greater benefit to higher ability pupils. 


Some teachers noted that scanning the live transcripts and pulling out the information required a high 
degree of cognitive processing as well as ability and technique in using tablets. Some pupils were 
also physically more adept at using the tablets and it appeared obvious to teachers which children 
were used to having tablets at home and knew how to use them. 


During the first stage of each of the phases, when schools were encouraged to only use captioning 
during literacy lessons, teachers reported that the captions provided important reinforcement of 
spelling, new concepts and words, and allowed them to give more detailed instructions. There were 
initial concerns about how captioning would fit into other lessons, especially maths (there were 
concerns that numbers would appear as words as would fractions and functions such as ‘three times 
four’). In practice, we had reports of captioning being used in several other subjects and having a 
perceived beneficial effect, including the comments below: 


e Maths—the layout supported learning (as it showed and therefore reinforced the use of 
mathematical symbols) and scrolling back helped children follow previous examples that had 
been worked through. 
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e Science—although children were working in groups and possibly at different stages the 
teacher could provide one set of instructions for everyone to follow at their own speed. 

e Physical education—during noisy activities the teacher could present their instructions to the 
class directly via the tablets. 


Overall, teachers reported the benefits of having captioning in class as being that: 


e Pupils could scroll back to clarify instructions or refer to previous examples. 
e Instructions or support could be given to pupils working in different groups at the same time. 
e Children needed to ask fewer questions for clarification. 


However, we found two issues that raised concerns about whether the captioning was having the 
intended impact: 


e Many pupils were not watching it in real-time, so any anticipated benefit from that element of 
the theory of change is unlikely to materialise. 

e Some teachers felt the processing required for children of this age to scroll back to find 
relevant information was so great that only higher ability pupils would benefit from this facility. 


Pupil surveys 


This subsection covers feedback on the end of lesson pupil survey. Generally this element of the 
approach was not seen as adding substantial value to the teachers’ feedback based on lesson 
transcripts, and yet reduced learning time in lessons. 


The theory of change states one of the approach’s outputs as: 


‘Students reflect on their learning experience and provide feedback to the teacher via a survey at the 
end of each lesson.’ 


The reason for getting pupils to complete a survey at the end of each transcribed lesson is so that 
they feel more involved in the learning experience and teachers are able to modify their practice 
based on their pupils’ experiences. In practice, the majority of teachers did not see any purpose for 
this aspect of the trial, and this was confirmed in our observations and discussions with the children. 


The survey contains five statements which are answered using a five-point scale by each group of 
pupils sharing a tablet. The statements include: 


e Please rate the quality of your learning in class. 
e My teacher explains difficult things clearly. 
e When lam confused my teacher knows how to help me understand. 


In general, teachers or leaders in participating schools did not regard children in Years 3 to 6 as able 
to review their own or their teachers’ performance over a one-hour lesson, nor to make valid 
judgements on a five-point scale. The overwhelming feedback on the survey was that teachers knew 
children would score them highly, just because they liked them. Most teachers stopped referring to 
this feedback as it had no relevance to the lesson. Most children did not appear to discuss amongst 
their groups what feedback to give and did not base their scoring solely on issues being asked about: 


‘Interviewer: So you just scored that one a 1. Didn’t you like the way you learnt in this class? 


Pupil 1: No, me and J[pupil] wanted to give a 1 last time but S[pupil] wouldn't let us ‘cos she had the 
thing [tablet]. So this week we get to do the scoring, so we’re putting a 1.’ 
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Completing the survey took time out of the lesson. In general, teachers stopped pupils working 5-10 
minutes early in order to answer it, and they were concerned that if this task was not relevant then 
learning time should not be taken up by it. 


Transcripts 


This subsection covers: 


Time needed to read transcripts; 

frequency of use of transcripts; 

use of transcripts for professional development; 
sharing of transcripts with peers and managers 


The full transcript of a one-hour lesson could be very time-consuming to read. Teachers were not able 
to find time to go back over most of their transcribed lessons, let alone review their performance. 
Inevitably, the more captioning the teachers booked, the more there was to read, which was a 
disincentive to using the system to its full potential. 


‘At first the only chance | got to look at them was at night when I was in bed. I’d read some bits out to 
my husband and asked what he thought! But now | don’t get round to reading them at all.’ 


Many teachers also felt that they were not in a position to necessarily learn much from simply reading 
the transcripts. Initially, some were surprised at how much they spoke, or noted how they could have 
phrased questions better and so on, but they needed external guidance on how to make changes to 
their practice. 


Fitting in opportunities to review practice, digest, and learn from the transcripts or feedback was very 
challenging. Likewise, we had very few reports of any participating teachers discussing their practice 
with others. 


The theory of change set out the intention that teachers will work with others, including their manager, 
to reflect on and improve their performance: 


e Teacher is supported to plan/set goals for their teaching based on feedback 
e Teacher discourse with colleagues around teaching practice is facilitated 


We did not find evidence that this was happening. In some cases where several teachers were 
involved in the project they might have had conversations about their transcripts or what they had 
thought upon reviewing them, but these were the exceptions. 


In general, we found very little wider discussion of the project beyond the specific teachers taking 
part. Interviews with headteachers revealed that many had not had time to see the captioning working 
in a class, or seen any of the transcripts of teachers’ lessons. 


This overlaps with another finding: that teachers seemed to feel more confident taking part in the 
project with the reassurance that all of the feedback on their performance was personal and so private 
to them. Many were comforted by the fact that no one else would see their assessment. 
Reassurances had been given by the intervention team that no data would be disclosed from 
transcripts, dashboard analysis, or the feedback report. 


Had the project required discussion of performance within schools it is likely that it would have been 
received very differently. Some interviewees mentioned that the system had potential to be used like 
an observation, so senior staff could assess teacher performance by reading the transcript or 
reviewing the analyses. This was perceived to be a daunting prospect and many spoke about how 
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they would see that as being counter-productive and that reviewing their own performance was more 
beneficial. This raises the issue of how teachers know, by themselves, how to improve their practice. 


The feedback reports were revised following the evaluation team notifying the intervention team of 
some teachers’ concerns. The final versions used contained a stronger element of directing practice 
for improvement, which may help address this. 


For any future trial it is important to note that: 


e Teachers did not have, and did not appear to be able to make, time to read through full 
transcripts 

e Teachers were not using their transcripts or other feedback as the basis for discussions with 
their managers about their performance and development 


If this system is to be seen as a tool for professional development, it needs to contain as many 
pointers as possible to how teachers can improve their performance. There should also be guidance 
provided to schools on designating some participants’ PPA time to review their performance on the 
project, and on encouraging participants to discuss transcripts/feedback with their managers. 


Dashboard 


Even though teachers were not looking back over their lesson transcripts, all regularly reviewed the 
data on the online dashboard. This subsection covers what aspects of this element they found useful 
and provides suggestions for further development. 


The dashboard contained lesson-level analysis of teacher performance, including: 


Teacher's average number of words spoken per minute 

Number of teacher questions versus number of student questions 
Percentage of teacher dialogue per session 

Responses to the pupil survey 


With the exception of the pupil survey results discussed above, teachers found the information it 
contained very useful and in some cases discussed their analysis with other participants. This was 
mainly as there was no guidance on what ‘good’ levels were and so they were comparing in order to 
get a sense of how they were doing. 


This was seen as a way of reviewing performance without having to read the transcripts and so the 
more information contained on their personal dashboard the more feedback teachers would be getting 
on their performance (if this is possible whilst maintaining the brevity of the data). 


Similarly to their lesson transcripts, teachers saw ‘their data’ as personal to them and were hesitant 
about the idea of other people seeing it. They did not like the idea of being judged by their analysis 
and some were initially unsure why particular features of their practice had been highlighted. 
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Figure 3: Example of data dashboard presentation 
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Feedback reports 


As well as reviewing the data on the dashboard, participants also found the feedback reports that 
amalgamated of this data very useful. This subsection covers what aspects of this element they found 
useful and contains suggestions for further development. 


The intervention team produced feedback reports for every participant at the midpoint and end of their 
trial involvement. The content and layout of these reports was amended substantially during the 
different phases of the trial, with later versions meeting the needs of teachers well. The reports are 
written by members of the intervention team based on higher level analysis of individual transcripts 
and changes over time. This analysis is carried out by comparing practice to an agreed rubric, 
covering 16 features of ‘good’ teaching (see Appendix 1). The reports covered teachers’ performance 
on: 


Promoting critical understanding and thinking 
Promoting engagement and understanding 
Providing feedback and additional Instruction 
Pupil feedback 


In addition, they provided information on how to understand the analytics in the online dashboard. 


The feedback report covered: 


Which aspects of teaching are covered by the analysis 

Individual performance in relation to the performance of other teachers in the same phase 
What the results mean 

Suggestions for improving practice 


The reports were seen as one of the highlights of the project. All teachers enjoyed receiving their 
personalised data and seeing it set within the wider context of good teaching practice. They all felt 
they benefited greatly from seeing their performance in relation to others and they hugely appreciated 
the tailored advice within the reports. There was a feeling that much of this background and 
benchmarking would have been useful for them to have known about earlier on in the phase, 
especially as it helped them make more sense of the dashboard analytics. 


In particular, the revised version of the feedback report provided to schools in Phases 2 and 3 was 
well received. Teachers thought this was a useful tool for them and an excellent way to review and 
modify their performance. They thought it was the right length and was set out clearly, including 
stating why certain features were highlighted. 


Initially, there were concerns about how long these reports would take the intervention team to 
produce and whether this approach was scalable to a larger group of participants. However, following 
the first round of reports changes were made which decreased the time taken to produce them whilst 
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at the same time improving their accessibility, without losing the personalisation which teachers 
valued. 


Following their experience with this trial we believe the intervention team are now fully able to produce 
these regularly to a larger cohort of participants and that this is a key element to the impact this 
project may have. 


An extract from a feedback report is shown in Appendix 4. 


Forum and newsletter 


The idea of an online forum to share feedback, exchange ideas for captioning use, and get advice on 
practice was good in theory as it could have facilitated peer support and the dissemination of good 
practice. However, as noted throughout, time was the major barrier to teachers using the one set up 
for this project. Forums do not work so well if there is little use and little information being put on them, 
so then participants use them even less. Participating teachers could not see how they would easily 
find sufficient time to be able to engage in this activity in a way that is useful for them and the project. 


The newsletter, on the other hand, was seen as a really useful tool and this may be a better method 
of information exchange (despite being less interactive). Teachers found it helpful to read about 
different approaches and experiences and the newsletter made participants feel part of a wider 
project which may have increased their motivation to stay involved. 


Perceived impacts 


Although no analysis of pupil performance or attainment was carried out for this trial, teachers were 
asked what effect they felt the system was having on their pupils. This subsection discusses 
perceived benefits and issues for further development reported by participants and covers: 


Pupil behaviour 

Literacy and use of transcripts 

Which pupils benefited most 

Professional development and teachers’ behaviour change 


A number of teachers perceived that the tablets gave pupils more focus, or kept those who otherwise 
would have been struggling to follow engaged. There were reports that, since having the tablets in 
class, behaviour had improved and there was less disruption. Some teachers also used the fact that 
‘someone else is listening in’ (i.e. the captioner) as a means of controlling behaviour in class. Several 
also made lessons they were planning to have transcribed free from all other distractions including 
having a sign on the door saying they were not to be disturbed, or limited the amount of small group 
work carried out (‘we dont do different group tasks when we have the captioner on’). These features 
could all have improved class behaviour. 


Teachers did not generally think that providing pupils with access to live transcripts was likely to 
increase levels of literacy, but did believe that it might help pupils to follow instructions in class and 
therefore to stay more engaged. Overall, the use of tablets by pupils was quite low and so the value of 
this (more expensive) element was questioned by teachers. If the captions do not act as a 
reinforcement of literacy and pupils do not regularly review them, or are unable to find the information 
they need, then the role of providing tablets to all pupils is unclear. As explained previously, teacher 
surveys were not felt to have been reliably completed, which also suggests the scale of tablet use 
should be reviewed. 


As noted above, there were differences of opinion about which pupils benefit most from the system. 
Some teachers felt it supported lower ability children as they were able to review what the teacher 
had said and get help from the transcript. Others felt that poorer performing children of this age were 
not able to scan and filter the information contained in transcripts in order to find the parts that would 
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be useful. They felt that higher ability pupils got most benefit from the system as they could make the 
most effective use of it. 


The age of participating children was questioned. Teachers wondered if even pupils in Year 5 (aged 
9-10) were sufficiently skilled in processing large amounts of written data and extracting the pertinent 
points, let alone younger pupils. They were also unsure whether having tablets in the classroom was 
adding value to the learning experience of children this age, especially at a point when paying 
attention to the teacher was being emphasised. 


Interestingly, not a single teacher had concerns about pupils being listened to by the captioner, and 
several thought it would have been helpful to have included pupils’ comments in the transcript. There 
were no issues raised about confidentiality as teachers felt that as long as individuals were not 
identified, this could only add to the value of the transcripts. They thought that there was more to be 
learnt about their practice by including children’s speech in the classroom and especially their 
responses to teacher questions. This may be a consideration for future development. 


The focus of this intervention, as perceived by participating teachers, does appear to be teachers’ 
professional development and improving their practice in line with evidence-based approaches. We 
do not believe that pupils or teachers saw the pupils’ involvement (either having tablets to read the 
transcript, or completion of the survey) as a crucial element of this. 


Teachers spoke highly of the effect the dashboard and especially the feedback report had on their 
performance. They were definitely more aware of what they were saying, how they were saying it, and 
how much they were talking. They were also modifying their behaviour in response to this. Certainly, 
after the six-week feedback report they had more of an idea of how they should modify their teaching 
and several examples were provided of steps they had taken. 


This element could be strengthened further by using the feedback as a focus for discussions with 
peers and managers. Teachers are always going to be limited in their development if they are 
reviewing their own performance and coming up with ideas for change themselves. Sharing the 
analysis and discussing ways to address certain features is likely to bring about greater benefit. 
However, there was great reticence towards showing this analysis to others. The reason for this was 
not clear, as it could have been a feature of how the analysis was presented or how teachers felt 
about being assessed by anyone. There was a fear that, if this approach was rolled out further, this 
could be the first step towards Ofsted inspecting teachers’ performance, and that others could make 
judgements on teachers’ abilities by looking at their transcripts, dashboard, or feedback reports. 


Very little was known about the project and the trial by other staff members within schools. A more 
Open approach to what is involved in captioning, the feedback and benefits for professional 
development may have helped to normalise the intervention, and reduce the worry. 


‘There’s lots of negativity from other teachers. They just don’t “understand” what we're doing. They 
seem to think people will use it to judge them.’ 


How much does it cost? 


The approach is estimated to cost approximately £367 per pupil, based on five hours captioning per 
week over one school year and class sizes of 25 pupils. This estimate includes analysis costs (£158 
per pupil), equipment (£12 per pupil), training (£125 per teacher), and live captioning (£192 per pupil). 
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Dosage 


The level of use of transcription was recorded by the intervention team (see Figure 4, below). The 
columns indicate the total number of lessons transcribed by all of the teachers in Phases 1, 2, and 3 
separately for the ‘pre-feedback’ and post-feedback’ stages. 


Figure 4: Number of transcripts completed in the pre- and post-feedback periods 


Number of Transcripts Completed in the Pre- and Post- Feedback Periods 
140 


@ Pre-feedback 


120 @ Post-feedback 


100 


Intake 1 Intake 2 Intake 3 


In the ‘pre-feedback’ period schools in Phases 1 and 2 were supposed to transcribe two lessons a 
week, while schools in Phase 3 were supposed to transcribe four a week. In the ‘post-feedback’ 
period schools were free to transcribe as many lessons as they wanted. The degree to which 
teachers completed booked lessons, and the degree to which teachers continued to use transcription 
in the post-feedback period may provide an indication of what level of transcription teachers found 
attractive. 


Data collected by the intervention team showed that teachers completed 65%, 70%, and 75% of 
booked lessons in Phases 1, 2, and 3 respectively. This suggests that in Phase 3 schools trialled the 
approach over a shorter more intensive period. 


In the post-feedback period, teachers generally chose to transcribe fewer lessons than in the pre-test 
period, although the average number of lessons transcribed varied between Phases 1, 2, and 3. In 
Phase 3, teachers were supposed to transcribe at least four lessons a week prior to receiving 
feedback. This seemed to encourage them to maintain higher levels of transcription throughout. They 
were averaging just under three transcribed lessons a week when free to decide their level of use 
(compared to one a week for Phase 1 and one a fortnight for Phase 2). 


Future use, sustainability, and roll out 


Anticipated future and wider use of the approach was dependent on schools’ experience with the 
technology, level of perceived benefit for both pupils and teachers, and confidence in use of their 
data. 
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In general, teachers were positive about the Visible Classroom project and were interested in using 
the technology in some way in the future, as well as recommending others to try it. There were some 
recommendations on how this should happen though. 


First, schools suggested that there be clear messaging about how the data would be used and they 
should be provided with reassurances that it would not become a tool to judge competency. 


‘It sounds a scary prospect to use, and no other teachers in the school are keen to try. They think it 
may be used to judge teacher practice. It needs to be made clear it’s for teacher benefit only, and 
cannot be used by, for example, Ofsted. Ideally it would work just as a teacher tool.’ 


Second, teachers were interested in seeing what impact the approach had on teaching and pupil 
learning, especially those with specific learning needs or who struggled with literacy and reading, 
before implementing the approach more widely in their class or school. 


‘| would love to use the visible class long term. | thought it was amazing, | really did, | loved it. But I’d 
want to see the data to check it is useful for pupils on a broader scale. The other teacher in our school 
also enjoyed using it but used it less than me. I’d recommend it to other schools for pupils with 
concentration, hearing, and behaviour issues. I’m not sure about using it with younger pupils because 
it may be useless if they’re unable to read sufficiently.’ 


Third, teachers who had used the technology thought that it was better suited to some lessons and 
subjects than others. The tendency was towards using it more for language-based subjects, such as 
literacy or history, rather than more practical subjects. This is important when considering advice on 
how teachers could best use the technology, and considering how it fits into schools. 


| would use it long term but only for some lessons and not all the time. | wouldn't use it in practical 
lessons though, because children need all the desk space and should be left to their own devices 
rather than distracted by the recordings. It might be better suited to use in the Autumn term, really, 
when we're less busy. This would give teachers more time to review and familiarise themselves with 
the technology too.’ 


Fourth, schools chose not to implement the project with their Year 6 pupils, or stopped using it with 
these pupils in the run up to the Spring/Summer term. This shows that they had concerns about how 
much it would support or distract pupils, or lead to deviations from what and how they had to teach for 
pupils approaching their end of Key Stage exams. 


The only negative feedback was around the level of technology reliability in the school setting, 
reported to various degrees by nearly all participants. Schools’ attitudes to future use were affected by 
issues with getting their equipment to work and the support and speed of response they got from Ai- 
Media UK. 


I'd probably not use it again and would advise other schools to ensure that all their technology will 
work every time before the class come in.’ 


In the case of the school who dropped out (and withdrew from the evaluation) they did not feel that 
this was a project they could continue with as it had not been adequately explained to them by their 
management. This should be kept in mind if considering a larger trial as participating teachers need to 
be fully familiar with the project, its ethos, the captioning, technology, and what its role is supposed to 
be in the classroom. Where staff are not completely engaged with every aspect of the approach, or 
willing to learn, or where staff change, this can lead to lack of fidelity or attrition from the project. 
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Conclusion 


Key conclusions 


Overall, teachers were positive about the Visible Classroom approach, and believed that it 
had the potential to benefits both them and their pupils. 


Most teachers were adept at using the technology in the classroom, even if they had not 
done so before this trial. There were some technical problems related to hardware, software 
and internet connections, but after an initial bedding-in period most were overcome. 


Though few teachers spent time reviewing the verbatim transcripts, the online dashboard 
and more detailed feedback reports based on the transcripts were seen as valuable tools to 


support teacher development. To maximise the impact of the feedback, teachers would 
benefit from being given greater opportunity to review and discuss their practice with peers 
and managers. 


Pupils did not seem to use live transcripts of teacher dialogue regularly, consistently or in a 
way that would suggest an obvious benefit in learning. Teachers had mixed views on 
whether the live transcripts might have additional benefit for disadvantaged pupils or their 
peers. 


Further research would be required to assess the level of impact the approach has on 
academic attainment. Prior to considering a full trial it would be valuable to undertake some 
additional development work to refine the approach. 


Limitations 


The main limitation of this pilot evaluation is that no explicit data was collected by the evaluation team 
on levels of pupil or teacher engagement. We did not monitor quantitatively how much the live 
captioning was used in classes, how much and in what ways teachers were using the transcripts or 
dashboard, or what level of discussion was taking place with colleagues and managers based on 
these. 


Moreover, no impact data was collected. This means we did not measure changes in teacher 
performance before, during, and after the trial. It also means that we cannot say if the approach had 
any effect on the academic attainment of pupils and, if so, whether it impacted more on any particular 
subgroups of students. 


More observations of captioning use in classrooms could have been useful, especially over time as 
the approach bedded in. Similarly, more work could have been carried out to explore pupils’ use of 
tablets, perceived benefits, and their opinions on the approach. 


Interpretation and considerations for any future trial 


Based on the feedback collected from participating teachers, there are a number of recommendations 
for how the approach, or aspects of it, could be developed for future wider use. In addition, the pilot 
has raised a number of issues to be considered for any future trial of the approach. 


The Visible Classroom approach was well received in general and teachers felt there could have been 
benefits of using such a programme. However, there was no consistent message on what effects it 
was having, which pupils it was most likely to be benefiting, or how it was likely to help them. 
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The initial introduction to the project needs to engage the relevant teachers directly, as well as school 
leaders, as this buy-in is important for the sustainability and success of the intervention. 


The training needs to clearly set out the theory behind the rubrics used to assess performance and 
the features summarised in the feedback, and why these are important to develop teaching practice. 


Schools should be prepared for the additional tasks associated with using tablets in the classroom 
and/or preferably have a staff member who is responsible for the setting up and closing down of the 
technology before and after lessons. 


Support from Ai-Media UK needs to be as responsive as possible and available by phone so that 
problems are resolved promptly. Where tablet faults are identified, replacements need to be in place 
quickly. More problems with hardware and software occurred initially and so the intervention teams 
need to be prepared for this or allow a bedding-in period. 


The dashboard analysis was the most highly used element of the approach and so could be 
developed further (to cover additional aspects of practice, or suggestions for practice development) to 
maximise the level of teacher reflection on performance and behaviour change. 


Feedback reports are an element of the project that could be developed even further in terms of 
coverage and signposting to support in order to add as much value as possible to teachers’ 
professional development. 


Analysis in feedback reports should be presented less as a personal tool and more as the basis for 
further discussion to aid development. At the same time, teachers do need the reassurance that their 
results will not be shared with anyone they do not choose to share them with. 


External factors, such as staff availability and SLT (senior leadership team) level of engagement, 
played a significant role in the implementation of the approach. The intervention team should try to 
mediate or allow for these. 


The theory of change may need to be revisited in terms of what are realistic expectations of an 
intervention in a real-world context. It may be that expectations of teacher performance and 
engagement, plus the anticipated level of impact, are too high. 


Future research 
There are a number of outstanding questions that may warrant further evaluation: 


e Which areas of attainment is this approach most likely to impact on and how dependent is this 
on the lessons in which it’s used? 

e ls there sufficient added value of having the real-time captions available to all pupils to justify 
the additional cost of the technology? 


There was little evidence of live transcripts being used consistently or regularly. In addition, teachers 
disagreed on how pupils should use the captioning, and about which pupils were likely to have 
benefited most from their use. 


Following from that: 


e ls the impact on attainment greater for some pupils than others? Is this likely to be more for 
disadvantaged pupils? Is this related to how transcripts are used? 

e ls one tablet per two pupils the best provision? Does this level of provision increase use of 
transcripts by pupils? How affordable is this? 
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The trial did not cover particularly remote areas, or those with known internet problems (such as rural 
regions). The unreliability of technology was cited as an issue and this is likely to be worse in less 
urban areas/less internet-enabled schools. 


There was evidence of a ‘tailing off effect over the course of the study. This reduction in engagement 
was shown by the reduced use of lesson captioning as well as increasingly reluctant participation in 
the evaluation. More work could be done to identify the best ways to maintain engagement over time 
and continue developing teacher practice. 


Finally, it could be worthwhile to investigate the relationships between student feedback and factors 
such as lesson content, format, or the teacher’s performance. 
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Appendix 1: Transcripts and transcript coding 


Ai-Media provided the University of Melbourne team with copies of the transcripts received by 
teachers (that is, the printed versions of the captions that appear onscreen during the lesson) to 
analyse for evidence of change in teacher practice or interactions within the classroom over the 
course of the term since the introduction of captioning. In order to analyse and understand effective 
teaching practice, a rubric-based coding scheme with four main components was developed (see the 
next subsection below). This allowed for an examination of teacher practice and provided insight into 
its potential influence on student outcomes such as engagement and achievement. 


Transcript coders were familiarised with the rubric and trained by the project manager who acted as 
lead coder during this project. Training included providing an overview of the rationale and decision- 
making process that underpinned the final 16 criteria, and a collective, collaborative analysis of 
transcripts, to ensure reliability across coders. Coders were then given several practice transcripts to 
analyse, which were checked and cleaned. Analysis of live transcripts began again once an 
agreement level of 80% or above was achieved between experienced and new coders on several 
practice transcripts. Finally, all coded transcripts were checked and cleaned by the project manager 
prior to being finalised and added to the database. 


Coders were allocated 15-30 mins for each transcript, depending on length. The transcripts were 
read and frequency counts were made against each of the 16 criteria. Relevant information was then 
extracted from each transcript and entered into a database. This comprised the frequency count for 
each criterion in the rubric; as well as the lesson number, type of lesson (e.g. literacy), duration, the 
date, the teacher’s name and intake (either Intake 1, 2, or 3), and whether the lesson occurred before 
or after the teacher received their individualised feedback report (i.e. pre-feedback or post-feedback). 


Teaching practice rubric 


Hattie’s (Visible Learning, 2009) work emphasised the importance of clear goals, learning intentions 
and success criteria; the linkage of new knowledge with old; clear instructions; feedback as a powerful 
tool with which students can confirm, add to, overwrite, or restructure learned information; a 
combination of deep and surface level pedagogies with questions tailored to both ends of this 
spectrum; and the use of peers as co-teachers. The most effective of these practices were used as a 
framework to develop a rubric of effective teaching practice, which was subsequently used as a tool to 
code transcribed lessons. In addition to the work by Hattie, works including those by Marzano (2009), 
Allen et al. (2013), Hamre and Pianta (2005), and the Measures of Effective Teaching Project (2010) 
were used to inform the development of the rubric, particularly for core aspects of teaching practices 
upon which more advanced practices hinge, such as managing student behaviour, creating a positive 
and respectful environment, engaging students, and so on. The rubric was refined over a series of 
iterations in which transcripts were practice-coded and the frequency counts were analysed 
statistically. The final 16 rubric items are shown in Table A1. 


Table A1: Teaching practice rubric 


Code Item Description 


Deepen Provides student with opportunity to deepen understanding and make 
understanding connections between ideas and build on prior knowledge through sustained 
context-specific dialogue and open-ended or divergent questions, to which there 


is no one correct answer 
Scaffolded activities | Provides opportunities for collaborative, scaffolded activities with their peers in 
and collaboration order to actively engage in learning 
Connections Help students to make connections to the real world 
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Convergent Checks fact recall by asking closed questions (characterised by YES/NO 
questions/quizzes), or questions which have one correct answer 


Review Reviews previously learned concepts at the beginning of and/or throughout the 
lesson 

Repeats comment Repeats comment or question from student before answering in order to 
encourage and reinforce correct answers 


Positive classroom Creates a classroom environment in which students have an opportunity to 
environment freely/spontaneously ask/provide task-related questions/feedback that is content 
specific 
9 Prompting Teacher uses prompting or probing (a question or statement used to elicit an 
appropriate student response) 


Summarise Concludes the lesson by recapitulating/summarising key points, AND provides 
opportunity for follow up/future engagement 


Behaviour Sets clear behaviour expectations and prompts behaviour reminders during the 
lesson 


Introduces and Introduces and explains new/complicated vocabulary/terminology; simplifies 
explains concepts by breaking into different levels or elaborates on abstract concepts 
using concrete, developmentally appropriate, and high-quality examples 


13 Resources Utilises written/visual/audio resources to support learning 
14 Important Emphasises important points 


15 Feedback Teacher provides immediate, specific, and corrective feedback to individual 
students/student group 
16 Goals/success Clearly expresses the purpose of the teaching content at the 
criteria beginning/throughout the class/activity (goals and expected learning outcomes) 
and refers to goals throughout the lesson 
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Appendix 2: The dashboard 


After each lesson, teachers were given access to an individualised online dashboard that presented a 
number of teaching analytics. For each lesson, the dashboard comprised teaching analytics 
(teacher’s average number of words spoken per minute, number of teacher questions versus number 
of student questions, percentage of teacher dialogue per session, and responses to the student 
survey) and captioning analytics (student’s rating of the captioning quality, number of unclear terms, 
technical errors, and overall audio quality). The goal of the personalised dashboard was to highlight 
meaningful information that would be useful for teachers in informing their practice and, also, to 
promote and enable regular, real-time, self-reflection on teaching practice. 


Figure A1: De-identified sample of the individualised online dashboard available to teachers 
after every lesson 


Analytics for session id: VISI1007D 


Teaching Captioning 

Teaching 
Average number of students in session Survey Measures - Rating out of 5 stars 
2 8 "Rate the quality of your learning in this class.” 

0.0 

7 = 

1star 2star 3star 4star 5Sstar 

Survey Measures - 6 point Likert Scale Average questions per session 


“The transcript from last lesson helped me with my learning.” 


i Really agree 


Wi Teacher 

BB Have not looked i Student 

at the transcript procedural 
1 Really disagree i Student 
i Slightly agree substantive 
i Disagree 
HB Agree 
IB Slightly Teacher 

disagree bod goats 

Overall average words per minute Percentage of teacher dialogue in session 


W@ Teacher 
dialogue 


No teacher 
dialogue 
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An online student survey was created based on the work of the Measures of Effective Teaching 
Project (2013). This work identified seven key areas that contribute to how a student perceives their 
teacher and achieves academically. These are the teacher's level of care for the students and respect 
for their ideas and contributions; behavioural control of the classroom; and ability to clarify and 
consolidate information, and challenge and captivate students. 


In addition to sampling the seven key areas listed above, the survey also examined students’ 
perceptions of the captioning technology. The survey was delivered via iPad to students at the 
conclusion of each captioned lesson in order to provide insights into the role of real-time captioning 
within the classroom. This survey gave students an opportunity to reflect upon the aspects of the 
lesson they liked most, as well as provide feedback to teachers on their teaching practice and the use 
of captioning. 
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Figure A2: De-identified sample of the survey completed by students after a captioned lesson 


ba) i live) 


How did your session go? 
Welcome to our survey. Please complete all questions and then press 'Send’. Thank you. 


Rate the quality of your learning in this class. * 


www 


Rate the quality of the captioning in this class. * 


www 


My teacher explains difficult things clearly. * 


Really ; Slightly Slightly Really 


| like the ways we learn in this class. * 

Really : Slightly Slightly Really 
— agree 
When | am confused, my teacher knows how to help me understand. * 

Really . Slightly Slightly Really 
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Appendix 4: Individualised feedback report 


Approximately every six weeks for teachers receiving a non-intensive version of the intervention 
(Phases 1 and 2), and every two weeks for those teachers involved in the intensive phase (Phase 3), 
teachers were given individualised feedback reports, as an aim of the project was to track changes in 
teaching practice after feedback. For Phase 1 teachers, this resulted in three feedback reports in total; 
for the other phases, feedback was given mid-way through and at the conclusion of the study. In this 
report, teachers were provided with feedback regarding their teaching performance on the 16 rubric 
items relative to the average performance of all teachers involved in the project. In the second 
feedback report, in addition to the group comparison, teachers were also shown a comparison of their 
performance in Phases 1 and 2 of the study. Teachers were also provided with a detailed explanation 
of the dashboard and given resources to assist in their professional development. These resources 
were individually selected in relevance to those areas of teaching practice identified as requiring 
future attention and support. 


For each captioned lesson, teachers received a frequency count for each of the 16 teaching practices 
(Table A1). For example, for a particular lesson, a teacher may have employed scaffolded activities 
twice, and fostered a positive classroom environment four times, and so on. This data was averaged 
across lessons, so that each teacher had an average frequency count for each of the 16 rubric items. 


Because of the relationships between clusters of these 16 teaching practices, four higher level factors 
were created, which were essentially composites of the 16 teaching factors. These factors provide the 
opportunity for clearer and more simplified analyses. All of the statistical analyses on teaching 
practices have been conducted at both the rubric level and the factor level, the latter option providing 
greater ease of interpretation. 


We examined the participants change their teaching practices over time (i.e. the 16 rubric items, and 
the four factors) in two ways. The first method involved comparing teaching practices in the initial 
stage, middle stage, and final stage of the project. Although, ideally, teachers were meant to adhere 
to a fixed timeline (that is, all members of a particular phase group starting and finishing at the same 
time and committing the same length of time to the project), this did not always work due to the 
realities of the teaching environment. Each teacher essentially worked to their own timeline (beginning 
and ending the project at different times and committing different lengths of time to the project). 
Subsequently, each teacher's initial, middle, and final stages were identified by comparing their first 
third of transcripts, to their second third of transcripts, to their final third of transcripts. Of particular 
interest was the participants’ performance on the 16 rubric items, and the four factors, across the 
three time periods. 


The major finding of these analyses, shown in subsections on rubric items) and factors, was that most 
teachers showed a peak level of improvement across a majority of teaching practices within their 
initial and middle stages of involvement in the project, but only teachers in the intensive group were 
able to carry this improvement into the final stages of the project. These findings highlight the 
important issue of maintenance of improvements in teaching practice within this project, which will be 
explored in the discussion section. 


The second method involved comparing teaching practices in the pre-feedback and post-feedback 
period of the project. Of particular interest were teacher’s performances on the 16 rubric items, and 
the 4 factors, across the Pre-feedback and Post-feedback periods. The major finding of these 
analyses, shown in subsections on rubric items) and factors, was that the majority of teachers showed 
a peak level of performance in the pre-feedback period of the project, and encountered difficulty in 
maintaining this high level of performance into the post-feedback period. The exception to this was 
the intensive group of teachers, who actually showed further improvement in teaching practice even in 
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the post-feedback period. This method highlights teachers’ differing responses to feedback, and 
barriers to engaging with the project over an extended period of time. 


In addition to analysing the rubric data for changes to teaching practice over time, we also analysed 
the student survey data and dashboard analytics, by comparing the data from the pre-feedback and 
post-feedback periods. We also analysed a small sample of the teacher survey responses that were 
available at the time of this report. 


The above analyses examined changes to teaching practice and student and teacher perceptions at 
the group level, with consideration to the different cohorts of teachers. In addition to these analyses, 
we also conducted teacher-level analyses in which each individual teacher’s initial third of transcripts 
were compared to their middle and final third. The results of these analyses, along with individualised 
teacher survey results.. The results of these individualised teacher profiles indicate that there is 
considerable variability across teachers, both in their initial teaching practices, and in the way in which 
they changed over the course of the project. Therefore, while substantial differences are noted 
between the three intakes of teachers, it is important to note that even within their respective cohorts, 
teachers showed considerable variation. 
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Appendix 5: Topic guide 


Making Teaching Visible 


Research aim: to understand how the project is working in participating 
schools as well as its impacts on teaching and professional development 
and pupils. 


The interview aims to gather participants’ views about: 

x how the project has been working in practice 

x ate) \ mm tat=m ©) Ke) (=1ex ima) oy= (eit lae me) aim (== (erallae nm ol6) 6)|(swr-lale maal> myer alele) mr-lale| 
x potential for the future roll-out of this approach. 


Introduction 


Thank you for agreeing to take part 
Introduction to researcher 


Introduction to NatCen—social policy research organisation, independent of Education 
Endowment Foundation (EEF), who are funding the evaluation as well as Nesta and 
University of Melbourne who are running the project. 


Explanation of research: 
NatCen have been asked to independently evaluate the Visible Classroom project 
The purpose of the interview is to inform future development of tool and project. 
We would like to understand: 
= how use is going and how you're getting on 
» what’s changed since the start of project 
» what changes could be useful going forward. 
Participation is voluntary 


What you say is confidential—we want to hear your genuine thoughts as this project is still 
being developed 


We will be writing a report of our findings but individuals’ names will not be included 


We will be recording the interview so we have an accurate record of what is said. Files are 
stored in secure folders in line with the Data Protection Act 


The interview will last about 20 minutes 
Questions? 


Ask for permission to start recording 
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START RECORDING 


Implementation 


Aim: to understand how teachers are using the system within the classroom and how useful 


they find the feedback features. 


Use of the system 


How they are using the technology and how this has changed since the last interview: 


Which lessons (format and content) 
How often used 
Use of tablets 


Experience of lessons so far 

How easily / quickly it’s fitted into the school / class 

Any times when they don’t / wouldn't use it 

Any novel uses such as using it in PE or Technology to display instructions 


Views about using the system 
Staff 
Confidence using the system 
Enthusiasm and engagement for using the system 
How feelings have developed since last interview 
Class, how pupils respond in terms of 
Attitudes towards / views of the system 
How it supports students 
How the above has changed since last interview 


Use of dashboard 


How often reviewed 
What they look for 


What’s useful and what could be further developed 


Use of scripts 
How often they review transcripts 
What they look for 
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What they’ve learnt about their practice 


Expectations of feedback 
Usefulness of six-week feedback: 
Level of detail, value of surface vs deep feedback 
Any issues highlighted 
What was helpful / less helpful 
How they have responded to feedback. 
Suggested improvements (the team really want to develop the feedback provided) 


CPD tool 


How they are developing their practice since the project started 
Use with managers and peers—development of CPD 


Other communications 


Use of forum 
Use of newsletters / sharing practice 


Perceived impacts 


Aim: understand teachers’ views of how the system is impacting on teachers and pupils 


On teachers 

Style of teaching 
Professional development 
Workload 

Effect on non-MTLV teachers 
Other impacts 


On students 
Ways in which pupils have been affected, impact on: 
Literacy 
Behaviour 
Attainment 
Engagement vs distraction 
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Lessons 
Consistency across all pupils, whether some pupils are benefitting and not others 
Other impacts 


Sustainability 

Would they use in longer term? 

Why / why not continue? 

Plans for roll out in school 

Encourage/ recommend use: 
For which schools/ types of pupils? 
Any other support needed? 


STOP RECORDING 


Reassure regarding confidentiality 
Check to see if participant has any further questions 


Thank for their time and ensure they have research team’s contact details 
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Appendix 6: Recruitment documents 
The Visible Classroom 


Project information 

Thank you for your interest in our ‘Visible Classroom’ project. This letter explains 
more about the project and what it will involve so that you can decide if your 
school is happy to take part. 


The aim of this project is to evaluate the impact of ‘The Visible Classroom’, a 
project using real-time captioning and lesson transcripts to support teacher 
professional development and student learning. The results of the research will 
contribute to our understanding of what works in teacher development and 
enhancing pupils’ learning, and will be widely disseminated to schools in 
England. The Education Endowment Foundation (EEF) have funded us to explore 
this project and develop it with teachers in preparation for a large-scale trial. 
We will provide teachers with a professional development tool to reflect on their 
practice and involve them in developmental research to help define how this tool 
is rolled out across a large group of schools. Teachers will gain experience with a 
new CPD tool to give them insights into their practice. They will also experience 
being part of a research project. CPD will be provided by the University of 
Melbourne team led by Associate Professor Janet Clinton and Professor John 
Hattie and informed by their influential work on ‘Visible Learning’. 


The Visible Classroom 

The programme is based on work carried out previously by the University of 
Melbourne. It involves the real-time capture of what the teacher is saying in the 
classroom —- a microphone relays what the teacher is saying to a transcriber who 
types the ‘script’ of the lesson. This script is used in two ways: 

1. Teachers look back at the script after the lesson as a tool for reflecting on 
their 

practice. They can also look at their scripts with other teachers and use them 

as a detailed starting point for conversations reflecting on teaching and 
learning. The teacher also gets feedback on aspects of their practice from the 
University of Melbourne (in terms of number of questions asked, rephrasing, 
use of open questions). In this way it may be a useful tool for continued 
professional development. 

2. The script can be instantly relayed back to the teacher and displayed in the 
classroom (on the whiteboard or on tablets used by each of the students) for 
pupils to refer to and recap key aspects of the lesson. 

You will be fully trained on how to set the system up and use it for both of these 
uses. There will also be support given throughout the trial for any issues or 
questions you have. 

Nesta and Ai Media UK are implementing the trial and providing support to 
participating schools. The University of Melbourne are providing the feedback on 
teaching. 


The development trial 

As this trial is to develop the system for a future full trial we need to regularly 
get your feedback on the programme, the IT system, the use of transcripts, the 
support and information you need and how it works in the classroom. EEF have 
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funded NatCen Social Research to work with us as independent evaluators so 
that they can find out what you think of the Visible Classroom and what 
suggestions you have for the future roll out of the programme. 

NatCen will come to visit you once or twice during the programme, possibly to 
watch the Visible Classroom in action and talk to pupils, and find out how the 
trial is working for you. They will schedule telephone catch-ups with you 
between visits to find out how you’re finding the programme, how it is fitting 
into school-life, and any impact you’re seeing on your development and teaching 
and the attainment of your pupils. 

The scripts will also be passed on to them to look at they type of feedback you’re 
being provided with. This level of evaluation will be the same in all 8 schools 
selected for the trial. 


The trial programme 

There will be two intakes of 4 schools onto the trial — one starting in Feb 2014, 
the other in April. The trial will run until mid June. Four teachers in each school 
will take part in the trial. You will be invited to a local 1 day training session on 
the programme and associated technology. You will then receive a visit to help 
you make sure you have everything set up correctly to carry out the 
programme. 

> Stage 1 - CPD as prescribed (4 weeks) 

e You will be asked to carry out 2 hrs of captioning (to produce the script) a 
week 

e This will be with literacy lessons 

e You will be asked to record how you review the scripts, if they feed into your 
discussions with colleagues and managers and any impact you see them having 
on your practice, pupils and school 

> Stage 2 - You decide (10 weeks) 


e You continue using the scripts for at least one hour of literacy a week 

e You will also have the chance to use the script and live feed in whatever way 
you choose, how ever often you like and in whichever lessons you like 

We will provide template letters for parents to let them know that your school is 
participating in the project and what it will involve. 


Outcomes 

Following this trial we will report back to the EEF on how it went and collate all of 
the feedback we have received from participating schools. This report will 

include 

recommendations for how the programme should be developed to operate as a 
full trial. 

We really appreciate your interest in helping us at this development stage, and 
the fact that you will give us honest feedback and opinions. This should enable 
us to develop the programme further so that more teachers and pupils could 
benefit from it in the future. 


Participating 
If you would like to be one of the schools to take part in this trial please fill out 
the form attached. 
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The Visible Classroom 


Agreement to participate in the Evaluation of ‘The 


Visible Classroom’ 

Please sign both copies, retaining one and returning the second copy to 
Oliver 

Quinlan at Nesta, 1 Plough Place, London, EC4A 1DE or a scanned copy 
electronically to oliver.quinlan@nesta.org.uk. 

School Name: 


Aims of the Evaluation 

The aim of this project is to evaluate the impact of ‘The Visible Classroom’, a 
project using real-time captioning and lesson transcripts to support teacher 
professional development and student learning. The results of the research 
will contribute to our understanding of what works in teacher development 
and enhancing pupils’ learning and will be widely disseminated to schools in 
England. 

The Project 

The project will involve a programme of professional development in which 
teachers use the Ai Live captioning and transcription system during their 
Literacy lessons. They will then receive a transcript of their lesson with 
feedback and use this to reflect on their teaching. This is a development 
project and their feedback on how useful the initiative is will shape the 
direction that the project takes. 

The project partners intend to cover the costs of the Ai Live service, feedback 
and CPD from the University of Melbourne and additional technology required 
to take part as part of the project. Use of some existing school resources 
such as broadband, PCs or tablets (if the school has them) will be required. 
We also intend to fund release time for the initial training day for all four 
teachers. 

Schools need to commit to supporting teachers in spending sufficient time to 
undertake meaningful reflections on their transcripts. Different schools will 
manage this in different ways, but for teachers to explore the potential of 
this project for professional development it is essential they spend time 
reviewing their lessons. 

This could take the form of some release time from lessons or meetings, 
dedicated professional development time, or support with tasks such as 
marking to reduce the pressure on their time. The shape this takes will be 
different in different schools, and exploring the best way to facilitate this will 
be part of the learning from this project which we value your views on. 

They also need to commit to the time needed to evaluate the effectiveness of 
the project, including communicating with NatCen to organise visits and 
conversations, taking part in interviews by telephone and face to face. 
NatCen are experienced in working with schools, understand the pressures 
on time faced and will work with you to make this process as convenient as 
possible. 
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This project is being run by a partnership between the University of 
Melbourne, Nesta (registered charity no. 1144091) and Ai Media UK. This 
project is funded by the Education Endowment Foundation (registered charity 
no. 1142111). 

We will let schools know if funding or support for the project is withdrawn, or 
if any partner is unable to fulfil its responsibilities under this agreement for 
any reason. 

The partners are not responsible for any costs, loss or liabilities incurred by 
the schools by participating in the project except as set out in this 
agreement. Any technology or equipment made available to schools must be 
returned at the end of the project. 


Structure of the project 

We will involve 4 teachers per school, ideally in two different year groups in 
key stage 2. Schools will belong to a ‘Phase 1’ group or a ‘Phase 2 group’. 
Phase 1 schools: 

wb 24th March - Training session for teachers (1 day) 

3rd March - 28th March - Stage 1: CPD as prescribed 

31st March - 18th July - Stage 2: You decide 

Phase 2 schools: 

30th April - Training session for teachers (1 day) 

5th May - 5th June - Stage 1: CPD as prescribed 

9th June - 18th July - Stage 2: You decide 


*These dates may shift depending on how the the project develops, but we will correspond with schools 
about any changes with reasonable notice and work with them to support practical implications. 


Structure of the Evaluation 

NatCen will come to visit schools once or twice during the programme to 
watch the Visible Classroom in action, talk to teachers and pupils and find 
out how the trial is working for you. They will schedule telephone 
conversations between visits. 

This level of evaluation will be the same in all 8 schools selected for the trial. 


Use of Data 

Data collected through the project will be used for the purposes of the 
project and for evaluation and research. Any data collected which identifies 
individual teachers or pupils will be treated with the strictest confidence and 
will be used by NatCen and the Education Endowment Fund for the 
evaluation, and by the University of Melbourne and Nesta for development of 
the project. The results of the research will be published, but no individual 
teachers or pupils will be identified in any report arising from the research. 
Transcripts will be anonymised and added to the University of Melbourne’s 
database on teaching and learning. Teachers who wish to withdraw a 
particular transcript from this database can do so up to two weeks after the 
lesson has taken place by contacting project lead, Oliver Quinlan at Nesta. 


Responsibilities 
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Nesta will: 

Deliver the training session on the initiative and the use of the associated 

technology 

Co-ordinate the implementation of the technology with Ai Media (the technology provider) 
Be the first point of contact for any questions about the project in general, the professional 
development activities and the evaluation 

Provide on-going support to the school in relation to this project 

Send out regular updates on the progress of the project through a newsletter 


The University of Melbourne will: 
Analyse the transcripts produced and provide feedback on features of teaching and learning 


Work with Nesta to adapt the initiative based on the feedback of teachers 


and 
schools 


Ai Media will: 

Provide training on the use of the technology as part of the training event 

Provide the equipment required for each school to make effective use of the real time captioning and 
transcription technology 

Provide the real time captioning service to schools for the sessions defined in this document and any 
further sessions mutually agreed with the school, Nesta, the University of Melbourne and Ai Media 
Be the first point of contact for technical support 


NatCen will: 

Correspond with schools to evaluate the project including visits, observations, 

interviews and collecting written feedback 

Collect and analyse all the data from the project 

Ensure all staff carrying out assessments are trained and have received CRB/DBS clearance 
Disseminate research findings 


Teachers will: 

Use the Ai Live system during their lessons as specified in this document 

Spend time reviewing their transcripts after their lessons and reflecting on their practice 
Consent to their transcripts being anonymously added to the University of 
Melbourne database on teaching and learning 

Consent to data regarding their involvement in the project being collected by 

NatCen for the purposes of their evaluation and report 


Schools will: 

Commit to carrying out the real-time captioning of lessons and the use of 

transcripts as defined in this document 

Support teachers in spending sufficient time on reviewing transcripts and reflecting on their practice 
Allow time for interviews and data collection as agreed with NatCen and as set out in ‘The Evaluation’ 
above 

Release teachers involved in the initiative so that they can attend the training 

session 

Ensure the shared understanding and support of all school staff for to the project and personnel 
involved. 

Be a point of contact for parents / carers seeking more information on the project 


We commit to the Evaluation of Making Teaching and Learning Visible and 
consent to the use of information collected as detailed above. 


Head teacher 

[NAME]: 

[SIGNATURE]: 

Lead teacher (main contact in school about the implementation of the project) 
[NAME]: 
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[EMAIL]: 
[SIGNATURE]: 
Teachers 
[NAME]: 


[AGE GROUP TAUGHT]: 


[SIGNATURE]: 
[NAME]: 


[AGE GROUP TAUGHT]: 


[SIGNATURE]: 
[NAME]: 


[AGE GROUP TAUGHT]: 


[SIGNATURE]: 
[NAME]: 


[AGE GROUP TAUGHT]: 


[SIGNATURE]: 
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Please note, the lead teacher may also be one of the teachers involved in the 


project. 
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medium, under the terms of the Open Government Licence v2.0. 


To view this licence, visit www.nationalarchives.gov.uk/doc/open-government-licence/version/2 
or email: psi@nationalarchives.gsi.gov.uk 


Where we have identified any third party copyright information you will need to obtain permission 
from the copyright holders concerned. The views expressed in this report are the authors’ and do not 
necessarily reflect those of the Department for Education. 


This document is available for download at www.educationendowmentfoundation.org.uk 


A Education 


Endowment 
Foundation 


The Education Endowment Foundation 

9th Floor, Millbank Tower 

21-24 Millbank 

London 

SW1P 4QP 
www.educationendowmentfoundation.org.uk 


